Standardized testing is prevalent in the United States today. Such testing is used for higher education entrance examinations and achievement testing at the primary and secondary school levels. The prevalence of standardized testing in the United States has been further bolstered by the No Child Left Behind Act of 2001, which emphasizes nationwide test-based assessment of student achievement.
At the same time, standardized testing is accused of a variety of failings. One criticism of standardized testing is that it can only assess a student's abilities generally, but cannot adequately determine whether a student has mastered a particular ability or not. Accordingly, standardized testing is seen as inadequate in assisting teachers with developing a level of mastery for a student in all subject matters.
Because of this limitation, cognitive modeling methods, also known as skills assessment or skills profiling, have been developed for assessing students' abilities. Cognitive diagnosis statistically analyzes the process of evaluating each examinee on the basis of the level of competence on an array of skills and using this evaluation to make relatively fine-grained categorical teaching and learning decisions about each examinee. Traditional educational testing, such as the use of an SAT score to determine overall ability, performs summative assessment. In contrast, cognitive diagnosis performs formative assessment, which partitions answers for an assessment examination into fine-grained (often discrete or dichotomous) cognitive skills or abilities in order to evaluate an examinee with respect to his level of competence for each skill or ability. For example, if a designer of an algebra test is interested in evaluating a standard set of algebra attributes, such as factoring, laws of exponents, quadratic equations, and the like, cognitive diagnosis attempts to evaluate each examinee with respect to each such attribute, whereas summative analysis simply evaluates each examinee with respect to an overall score on the algebra test.
One assumption of all cognitive diagnosis models is that the assessment items (i=1, . . . , I) relate to a set of cognitive attributes (k=1, . . . , K) in a particular manner. The relationships between assessment items and cognitive attributes are generally represented in a matrix of size I×K and having values Q={qik}, where qik=1 when attribute k is required by item i and qik=0 when attribute k is not required by item i.
Using the Q matrix representation, conventional approaches to cognitive diagnosis were developed to either (1) diagnose examinees, by assigning mastery or non-mastery of each attribute to each examinee, without determining the cognitive structure of the exam, or (2) cognitively evaluate the exam, by statistically evaluating the relationships between the items and the attributes, without diagnosing the cognitive abilities of the examinees. If items are not cognitively evaluated, a cognitive diagnosis has little meaning except with relation to the particular assessment examination. Likewise, if cognitive diagnosis is not performed on examinees, a cognitive evaluation of the items cannot be aligned with the observed examinee response data. As a result, the interpretation of an evaluation of the assessment item does not relate to examinees. Moreover, in order to be useful for evaluation purposes, the parameters of a cognitive diagnosis model must be statistically identifiable. No conventional method incorporates all of these requirements.
What is needed is a method and system for performing cognitive diagnosis that evaluates both assessment items and assessment examinees using statistically identifiable parameters.
A further need exists for a method and system for evaluating assessment examinees with respect to a plurality of attributes based on responses to assessment items.
A further need exists for a method and system for evaluating whether assessment items assist in determining one or more attributes of an assessment examinee.
A still further need exists for a method and system for evaluating an assessment examination based on the responses of assessment examinees.
The present invention is directed to solving one or more of the problems described above.